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ABS^BACI ■ . ,. ■ ■«-•■■. • ..' ■■'„'',■•': 

/ The conceEn with cospetency-based ed^ucatioa is hpt. 
HhetheE it is any diffeEent £eob the soEe tEaditdio^kL appEoaches bot - 
wh<«theE it is ^oEth the' considerable effort it iavolv^m. Xhere are 
several aspects of a cospetencyrbased prqgran\to be copeidered in i:ts 
evaleiation. Th^ first is whether or not there ^ a jastifiablf need 
fot the sp'ecified cospetency. Secondly, the earlcet, if any* avst^ be 
identE.ified« The testing of the prograe or specific coapetency avet bto 
cbnsidered. to assure that the test 'popalatioh is coapasable to tibe 
popalAtion for which the prograa is to be used. Ihe progsaa shonld 
coaparld for effectiveness to the alternatives on the edacatiopal 
aarket. it.aast be tested over an extended period of tiae t^o 
deteraine it.s l;oag-range iapact • An e valuator of coapetencybased 
prograas aust^. also detiaraine if there are any undesirable side 
effect»*Qf the prograa and whether the process itself b^s any 
daaaging attributes, it is 'also necessary to determine whether tjie 
positive results ^f a field test are actually due ta the prograa or 
to ^oae outside factor. After evaluat4.on of all the aspects of the 
prograa and its developaent* the prograa as a whole aust be 
considered in li^t of its cost to see \if it is a cost ef|ective- 
iaproveaent for ^he schools. (BBD)' <. ° 
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or, If the program is cxsigpetency-based how cane 
/' the evaluation is costing, so inufih? ■ 



1, Intrcduction / . ' . . „ 

A CB program is presumably thoroughly developed by tiie tiine it is in fiUl- 
scale operation, which presi^iabiy means tliat it gets its. graduates up to design 
level of quality and quanti"^ of achievements. . (We pass lightly over the fact that ' 
' ic my do this by. lowering the design level in the light of realistic field trials.) 
rie>e at last, one feels — or is it just that one hopes?— is a program that knows 
whar it does and says what it knows. Here we have laid to rest the . misleading ' 
. advertising of inflated phrases and vague promises that pervade traditional i 
education catalogues. ^ ^ ' ^ 

Apart from the irioral and scientific charniis of the CB approach long extolled" 
in' prose and poetry (soJ^ehow the poetry seems to creep bafik into the catalog^ 
there does seem to be at least one obvious peciiniary advantage al)out this 
■ innovation,^ namely the elinuj:iati9n of any need for fancy- evaluation. - • ^ 
C3E, the' honest fellow, wears liis— or is it her~credentials on its sleeve ^ Jh^y 

; • • . ■ . ■ , . . ■ ;v ■ . ' ■ ■ , 

need no" embellishment'. ■ / 

' ■ . I wish I could add to your enjoyment, of this evening's banquet by 
encouraging you* to believe that the CB'^^th is indeed "rewarded in this way, but 

. I propose instead to contribute to indigestion by stressing that the direct 

cost and quality of evaluation in CB programs that I 'know about is, not even in the 

. ballpark foip adequacy. Notj I hasten to add,- that . this puts it in a poor • — 
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position coinpa;>£d to luo^t educational programs. While it is true that ."tiTe CB 
approach offers the possibility of a nodest saving in one dijiiension of evaluation,, 
this/saving is usually not realized or not realized fully^ and CB evaluation 
desig^o often leave unanswered several fundainental questions about the,;^[nerit 
of the program. My chief interest liiis evening is to outline the basic features 
of a satisfactoi^ir evaluation desigi. 

To' ensure a sweet aftertaste , to ^his rather .bitter post^<x>ffee corxiial,, I 
shall add a word.dr two about the cost^. of evaluation , directed to^ the thesis 
that the median net cost, to /the appropriate agency , of a thoroxagh evaluation, 
is. negative. 



2The Main Problem . ' • . . ^ ^ „ * 

The mail} problem with the CB movement V, as I see it today, ^is not~as - 
the "sideline cynics are always saying about innovations — ^that it does the same 
^Id things and just gives them a different-name. There really .is a different ' 
orientation and a different methodology as weLU^, a' different vocabulary. The 
point can be quite 'well-illustrated by reading the program for this meeting. . 
It evidences no mere rechristening activity, but an effort-by many participants — • 
to focus and explain theii\aims' more clearly, an. effort tiiat definitely affects 
the content of at least some w -die presentations. , . 



"1 



The main problem, however, is still something pretty sijiple. . It is the 

" ■ and 

problem whether the total effect of the new orientation / methodology is 

worth the very considerable effort it has involved, especially in light of the 

dubious justiJpication of many of the; eonnpetencies. aimed at. Ihe aim may weli 
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7 •. • . ■ ' :y ■ ■ . ■ ■ ■ ■ ■ ., y'„ / 



be sucQisssful, ^utr it's^no^tf^^ allciear it "was .sensible. 

The arialo;5y with thfe beh4vioui»l objectives and crdt 

testing . /*'°^?i^S£%lly'sp2^ings- to mind, >as I travel around the • • 

mjor nietr^ipolitan school'' districts talking to school principals and disl3?ict 

or state staff, I, believe I have noticed a steady deterioratioin in the . . 

interest in what ]; call the credentialling of objectives or criteria at the 

expense of a pro forma interest in BOs or criteria as such, or perhaps an interest 

■ ' . ^ • - ,. ■ ' • ■ ■ 

in theiFi for wJxat they contribute t o ^conceptualizing ' the educational activity ♦ 

whetrher planning, perfqpning, or reporting it. (Cf . also^progronming of texts) 

Precisely the saxae problem— as is no-doijbt obvious to most of yC)u~f aces the 



'Hioveinent. 
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CB . . I tbegan with a no-nonsense orientation towards pragpiatic teaching 



and evaluation, with enphasis on pay-off skills. . But pretty soon we begin to find 

! V * . original identification as an 

that the concept of a coirpetency floats free, of the indubitable necessity 

for satisfactory job-performance, and is taken to include all sorts of. juggling acts 

that someone or other has takWi a fancy to. These' ^3?e mere process conpetehcies, and 

. efforts to disseminate them in the absence of prioof that they're iiit)drtant f or 

"..■•/•• 

pay-off performance represent exactly the kind of faddist, ill-grounded approach 
which the CB orientation was instigated to avoid. ; Only an uncompromising commitment 
to systematic and comprehensive evalxoation of each program can avoid this kind of . 

• : ■ ' - ." ■■• ' ■ ■ •■■ • A- ■■ 

rot from underroining what could ^^x>yide the fbundations of a better appix>adi to. . 

teach^ education and to many other kinds of ediKnation. I'm deliighted to see 

symptoms of concem with this ppint in. some speaker summaries, e.g. N. L. Gage's 
• ■ ^ ■ fir^t ^ .• ■ • . ^ ' . ••: ■ 

reference to the need for^establishing desirability in teacher behaviour." It's . 



. equally conspicuouk by its absence in iiiany other 'simffiapies. Perhaps that *s. 
' just because* they, ai?^ suinmariesi" we stell all find out soon. 

. 3> l^he Basic Requirements for Good Evaluation 
Let njs try to provide, in checklist fomy and a^ concisely possible, an 
outline of the points, that jnust be covered^^in the ev^u^iqn of a competency- 
based teacher education prograin. They deriv^* from a general checklist, which 

. . ■ ■ ^ \ ' ' ■ ■ '■' ■ ■ ■ 

was originally developed for the evaluation oA educational products, but: which ^ 

. .has since been expanded to apply to prograims.ana^^ . (as well as to evaluatofs' 

and proposaJ.s'V etc. ). /"Hiis presentation will- avoi^ the illustrative but sligiitly. 
irresponsible -anecdotes u$ed in thev Houston talk, where we were facing all the 
prbbleiTiS or after-dinner speeches, and 'it will .omit the details of rating scales. 



■ etc. 5 which are required in order to ijnplement .a checklist evaluation in practice. 



7he intention is simply to pl^y out the KanjTdijijensions which have to be "eiq>lored 
. in the course of evaluation, and which are not met by the mere wavixig of the nagic 
wand on which are inscribed the words' "competene/-based".\ 



5,1 'Need 

■ .Ihe first question, of course, is the ^ne that I \ mentioned a little ^earlier— the 
* question of the justification for .this particular -<»Tpetency,\ set of oompetenci^^ 

programs coirirdtted to teaching us -fliis set of conpetencies, etc. (For convenience, ..' 
- 1*11 focus on a component of - a CBIE project, which isNdevoted to teaching a particular 
^ competency.) Tne problem of credent ialling ahy successful training program is 

essentially th^ problem of hooking up its products 'to a needs assessment^. Can we * 

siiw that the need in 'questic^ is a real need? Needs assessments are typically 

-. • ..... ... \ ; ,. 



" done in two completely spurious ways; eithfer by using \standardized tests and 

•v ' ' ' ■ . ■ .... J '\ ■ . ■ 

assuming that short falls on these tests represent need, or by doajig a survey or 
■ * ■ ■. ' ■ ' " . . , ' .. ' ■ . t . ■ 

^•*-he population that will be affected or that is thought to be the population that 
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Lll be .affected) by the .-training program* Ihe deficiencies id.th each should be 



obvious,, .although both are pr>ovided examples in current "textbooks devoted 



in whole or in to program evaluation. There is no: need in the \iK>rld to, be 
■perfprmng' at nom level on any otand^rxiized test. For that matter., there isn't 

any heed to be.perfdming at norm level on any criterion-referenced test, unless """^ 

. . •■ ■ ■■ ■■• I . ■ " / " '■ ■■ ■ ■ • ' ■ ■ 

the criterion can -be demonstrated to concern, e.g. survival or ability to cope • 

with further e5q>eriences that mil be leading one towards, an, adequate collection 

of survival skills, ^.or -etc. Given the differentials in rat^s andidirections of 

ir-aturation, it really iwouldn*t Mke' iijuch sense to say that scat^Ettx^y had a need :to 

^ierfonn at norm level (mastexy level) even on a criterion-referoiced survival- 

connected curriculum except at the exit- test. ' V . , 

' * "-t^ ' ' . ' . ; . . 

Ihe d^iciehcies 'in the ."usual survey approach' are eqvaallV serious ^ People are 
very poor judges ..of what they really need (as opposed to^what they want ) where the 

issues^they are passing judgment on are either 'technical or complex or emotiorial - 

* • - .-■ •* ■ . ■ . ■ ' • . ■ . ' ■•■ -. ° ^ ■ . ' ' 

or novel. Since nost of. the areas where the surveys are taken feohcern issues 1±at 
rfie^ not just one but 4ll of these conditions , they ^are staggeringly unreliable 
guides, to need. I^jTi'not suggesting for one jxment that there is some omniscient 
entity around who can sixnply an^ reliably identify the "true needs" of populatiais 
of people;. People may well be 'the best judge; but that doesn't .mean that they're .. 
the best judge in. the short run, when uninformed. about the dssue, and wh^^^^tom • 
by conflicting concerns Over it. There are of course . ccoeas in teacher education 
where these problems are npt serious; but if you start looking into the question 
of values clarification in social studies curricula, citizehship education, sex 
education, propaganda analysis, ecx)logy, race relaj:ions, career choice, sex role, 
and a dozen oOier exa(nples from the everyday activities of teachers in . the school 
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classroorr*s of 197^4, then you can readily see that the survey approach to needs 
assessHient is likely to be Unreliable. I'in nc«r here to offer solutions — ^a solution 
t6'this prbbieni requires soiie specification of the exact version of the problem 
that is facing; one at the particular luoir^.t — but only to indicate the nature . 
and dlTiensions of the problem, and I'll leave this point with the single thought ^ ■ 
xriat needs ass^essments of suitable degrees of, reliability can indeed usually bfe 
Tdone,' but it -takes a lot of work and thought to do it." 



• , ■ 3.-2 Market ^ * ■ " 

." • ^ " '^^z ■■■ . ' •■ " •■ ■ ■ ■ . ■ . '* • . 

Vihat is needed is often unniarketable., and what is marketable is often r\pt needed; 
once the distinction has been reo£)gnized, it's clear, that we have to inves-feigate 
iiiarket separately. frpin need, or else we will.be left with a training program for 
which the' development ironey will have been wasted, ■sinc^ it cannot be disseminated 

-iro any signifiqant /number of the. centers. NIE is in the process of rethinking- its 

•■ . • ■ ■- . . ■ - * • • 

corr.rdtments in the direction oS dissemination, and this is a most welcome nbve, . 

■ since there has been a long tradition of assuming that dissemination will take, 
care of itself, an assumption which shows' optiMstic riather than a realistic 
natiare. ' , " ^ •■ ' ' i . .\' • . . . 

■ 3,3,. Perfgnrance Data — True Field Trials?, ').:■'.:•: 

■ Bo .we have a training program that really can deliver the promised competencies?.. 
Often, we have one that has operated very successfully in, ah "intensive care", 
situation, guided by -the inventor's hands and .enthusiasin, fiut never tested out : 

^ in the field, where it's^ot, to sink, or swim by itself.. Wher^'there have been 
field trials, they've often been semi-subsidized, with materials or assistance. . • • 

■ ' , ■ y ' ' ' - • / ' ' • • /' ' . ^ 

provided free of charge and that welcome little ^feeling , of being part of the/ 



in^group being supplied vjixhout" any charge at all. As- lCSig'as you can coast along- 
on th^^i^wthoirne effect, that will vk:>^^ splendidly; but it's a "ciisservice to everybody 
to suppose, that you can disregard the extent to which you owe your, success to that . 
handy Ixtxfe; booster. , 



• • 3.^" Performance Data—True Cohsumsrs? 

Who have we looked at as the ^impatted population for' our 'little .educational pack^e? 

•• ■ ■ ■ ' ■ ^'h ■ ' •. • • 

— ■ ' '■ m 

Obviously,' its primary impact (chronologically, at least). is on the teacher trainees 

who are going thrpiigh the prograins. A good evaluation doesn't rest on its laurels 

afxer looking at the : first contact. After all,, the justifiopition of this little . " 

nbcule is meant, to lie, not in its beneficial effects on teachers, but in their ' 

boner icial effects on their students. So we nor^naliy^ would need to look into 

rhe question of whether we have data on the eventual student effects. The. .widening . 

ripples do not ce^se at that point, however; do, we have data on inpact on future 

en^ployers*? t^&iat aboulEiim^ 

■■ , . ■ - , ^: ■^-•■----■^-.-^.....^^^^ ■ I ■ . 

activi-cies of the firs;t Ifevpljof students, i.e..,' tiie teacher trainees? What . - ■ 
about special derrands placed rbn the trailers themselves-'-have we investigated 

, : . ■ ' ■ •■ ri:- ■■ ■. : , . . ;■■ ■ . ■ ■ , . , , 

them as an iinpacted ^udience? The real consumers are a much less elite group 
than Ki^st evaluations suggest. And the news from the distant shores to which * ^ 
the effecrs finally spread is often not as good ash:he news, from the points close ■ ' 
LOthe point of first impact. . 



3 . 5 ?erf orrrance Data—Critical Competftors . , " - v -^'t ' 

Useful evaluation is always — implicitly or explicitly--coniparAtive. The real' 
' question for^ the. person vho is thinking abo^f adopting or purchasing a competency- 



. •■ -N- ■' " , ■ • ■ ■••*." 

J»sed program. is whether it is better than alternatives, rjlot whether it^ is better . 
than nothing at *a^-l% — -Herice,. the evaluator sets up, 03? explores EKIC,- .for evidence 
about criticii competitors,. The choice is hot between nothing. at all, biit 
- between actual competing products or programs. r V- 

So, the choice is between a competency-based program and an existijig program, or 
' between the competency-based program, and an alternative ^ innovative program, e.g. , • ' 
apprenticeship systems with on-site workshops,* etc 

Now, in the vanguard of new irovements, th^re is always consideral^e resistance to 
having a new iibvement comparad to the old one. ■ The avant garde ■ of ^he new curricula- 
projects' of' the last decade were ^always talking .'about how inappropriate it was 

■■ • . . ■* . ■■■ . ■/-' . ^ 

to compare them with the old ones . ■ The legitimate part, of ' this \/j>^int was that it f 
would have been unfair to compare them with the old ones, using only the. old ones' 
criteria for the standards, of comparison. Kowever, it's not at all inappropriate 
, to o:>mpare-the new ^id\the' 'old on standards that embrace thQ best' points of each.. 

It is a matter of considerable significance ^to discover that Japanese children out- 
: perform, their Arerican age-peers, both on the Japanese curriculi^k^tests and on the 
' American curriculum tests. . Whatever the'' explanation of this is, and it 'isn't . • - 



necessarily a sigh of the failure of the AT*er?ican school, ther evalxaator is making 
an .^xtramely useful-vcontribution "wnen 'he' or she identifies 'this difference. ■ Now 
you .canub:;t your life that canpetency-based "approaches always have beenv compared . ^ 
■ to traditional approaches on at least one dimension--namely, cost. AndNthe f act ' 
is. that it's compajped Qn.^a great many other dimensions too; the- extent of \^sruption^. 



opposition by itaff,' e;tc. , etc. -It's a nice, comfortable little insulation' move • 
to suggest Wt . the newi is only to be evalu^^ But if 's totally 

tjiipealistic, and it's illegitimate. -It took niany years for economists to.discbver . 
that-to'. their embarrassment-a cojnpetitive market for refrigeratdrs- embraced not 
. only refrigerators- but dishwashers and electric stoves . That is , tiie^ rational 
consuner- quite proper^ly (and not . irr^tiohaUy) Weighs the purchase of. a new ' . 
applian-ce .against the,,purchase of other possible new appliances. , Well, so it . 

mxi^/Q^erx^^^ to earn its spurs;, in the fr«e-fbr-all 

of the. educational marketplace, not ; in : some 'sequestered cloister. .. . . ' ' . 

-. . : 3 . 6'- Performance Data-nLong^Term V ; 



We're very Lnclined to suppose thit the termination of treatm&nf is the appropriate 
tijre to measure the effects of ^triatment; and we thereby confuse logistical sinplicity 
witr- methodological propriety/ It's exceedingly inconvenient to have °to segregate . 
f-unds for -use in long-terai funds i but it's exceedingly unreliable to, act as if - 
= soniehow xhe results of 'such a follow-up will automatically .-'bf tesame-as the 
results of an . eyal;aat ion at. the terroihatipn of treatment. 

3 .7 Performance Data— Side >£f facts . - 

■ Kas there.been.a. systematic, scientific, and sophisticated sear^ /f or side effects? 
It has. to be-_bon.e, iii^mind that from tl.e -point of ..view of the . evkuator , the goals 

■ of -e^e ' program, are really not . a Wtter of great iiportan^^ ■ 
interesting for "the archivist, .for th^ developer, and for "the fundli^^ agency. • 

"But what interests an evaluator is what the ;progr^ .actually did, juWd by its 
^:congruence with-the- -needs of the impacted 'po^^ It doesn't reaoWrf^^^ 
much.v^ether what it did was, what it meant to jdcX^hether it f^led tVdo what 



it Jiieant to do; but did son^thirig else instead, etc. , etc. It's up to -bie - 
evaliaatop to find out exactly what it did, and of course this means a great 
d^^al^of fishing '^ii).^rrto^ waters where no guidance is jartovided by reference to 
goals Stat eJtents. The search is not altogether without its own attractions; for 
the mur]^ waters of goal staten^ts, frequently inconsistent or incompatible with, 
1fhe goal statements of others associated with the project, or put of date, or • 
'based on factually false assuirpjions , etc. , are- by no means obviously more 
attractive than tho^se whdrein side effects lurk. . V • " ■ - ' 



3. 8 Performance Dat< 


i-- Process 




■ ^ 



It's quite wrong ^to use process^ as a reliable guide to. QUt^^ but that doesn't 
mean that there aren't legitimat^ reasoris fdir inspecting process in the course of 
"evaluation/ Tour reasons are . ■ . ^ ... 

a) the search^ for injustice. Whatever the outiaries, -^ere are certain 
types^of procedure that are ii^apprppriate for itoral reajsojis, and inspection of. 

process must be nade to see whether, excess cruelty , iiiequitable nethods of grading/ 

\' • • ■ ~" \ ■ ^ \ ■ 

\etc. ,, are iJi$>ort^:t. . ' \ ' \ . 

■ ■ ■ ■' ' • . . ' '■. ' \ ■■ ■ ■ ■. ■ " " I • - ■/ / * 

b) . the search for joy. liie second question is whether there is pleasure 

in the. learning fej^jerience. There are^ other reasons for jtxying to^mate learning^- ■ 
enjoyable, namely, that to dp so i)^p3Lil<ely to increase tip probability of learning 
taking place. But if all: else fails,' if .learning gains ire negible and^if tiie cost 
is co^^ ' then we might as well take account of the question df whether. iJie ;_. ^ 



• babies enjoy .baby-sitting of type 1 more than they do type 2. ■ I'd say that joy ^ • 
is a secondary end of education (because if 'it's joy you want, th^ .for ifost people' 
education l^as seine overwhelmingly powerful cf^itical coii5)etitors) , but that doesn't 
irean it sihouldn't . be an aim. 

O . ^ c) ; characteri?ation of the process .\ When the evaluation is done, soiebody- 

JC , oA_ >:,■-• • - ' ■ 

gpiiJg "^^^^^^^ i)ackage that was W a labeit^ 



When it arrived at the evaluator^S'^doorstep, but 'that label is as often as not 

^ ' tf*- ■ • ■ • • ■ 

extreniely irasleading;. ^d one of the tasks of the evaluatpr is to decide ■ whether ' • 

this i:;deed deserves the title of "ccqpeten6y==basea" training or not." 

c ' ■ • . * . , • , r ^ ' , ' *• - . "'• ** 

. i-d) causation. '^Insoect ion of process oft'en^ provides valuable -clues' '^^ 

> ' • ■• : > ■ ! ■■ " ^ • •< ' ■ ■ ' ■ -'.I • ' - • . ' ^ ^ ■ ■ -.^ 

to deterirajie the responsible agent for. any changes that are observed. Iji particular> 
it teiy be necessary for you ' to .inspect process in order to- settle the question 
of Whether \the alleged cause was the real cause^ of the changesy in situations ^where 



/ 

control; groups and baseline data are Inpos^ible. 

3.9 . PerforriBnce Data— Causation 



Expanding ^on the last thought of the previous subsection, we have to face the If act that 
alriKist every evaluation constitutes.. .an, invest i^t a causal claim, i.e. the 

aiiTi xhat^such-and-such a procedure had siicii-ahd-such^ (allegedly beneficial) effects; 



/-one"*of'^the reasons^ why" evaluation reseai^ch.is by no ineans an easy type. of. research. ^ . 
■.Quite.. often in teacher ^education, -the an^ount of .learning that" takes place from peers 

is aistii.culy comparable* to .that which occyrs from the^ training process itself ; " 

and the san:e. applies to "natural'Vlearhi^.frbrn field e^Jperiences as opposed to 



the contribution of the supervisor. ", ' v^^ ' ' ■ jlf^^;..^^ 



; : ■' ■ ■. ■ ■ \ .- :■. . ^ 

3.10' Performance Data — Statistical Significance^. 



/ 



\ 



It is' scarcely necessary 1;o- mention tha-t evaluatlpn frequently requires some 
razhpr elaborate statistical investigations, in order to "determine the true 
sigTAificance of an apparent effect; or •'indeed in order to identify any effect 



whatsoever^ j; 



■ V 



.'• ■/ ■ • ; ' ' ■ ' ' ' ■•; -12- ' . '. V 

. : / . ■ 3,11 EduGA t5Dn^l Significance . ' " • . - 

'■ When v/e are iri possession of reports on 'the first ten checkpoints , we are in a position 

to ;tie them , together and' con;e up with a.. conclusion, about -educational. significar4ce. 

^ ' • ' " - ' ■ . ' ' ' , •. 

■ All too often, educational significance is identified with statistical si@:iifxdanc^' 

/• ■ ■ ■ ' ■ ■ ^ • I . ..' ■ ^ ' •/ 

7 Ox th<s difference between the treatment grouo and the control group. But of course 

■ ■ ^; ^ ' / • • • . ■ • 

. / .. that! air ferende' riicty lie* due to the:use "Qf avery larg^^uni)er of^ubjects (wl^ich 
/JS^ r/iagnifies^ trivial, effecfs) , to inyalid- tests, or to^a dozen oth«j causes, none .of 

,y which "represent any -contribution to e^iucation. .Chec-kpoijit 11 is thus a suinming-up • * . 

■ / ' " '• ' . .•■ : ' ;^ / . ■■ ^ ■ . •• 

■ / chdci<point, wher^ the results of detailed item analysis; -niay' come . in, where inve§- 

/ "^ tigatidns of instrun;ent reliability' may come in, and where it is frequently -necessary . 
^uo-rafer back to one's needs assessment data, in the ligjit of one's refined' " 

.■ c?aaracterization 'of .the product,- to- see whether ih. fact the product as it turns '/ 

' ■ ' ' • .\ ' ' ■ . ' . , ■ ' ' V •.. 

out ta be really meets, the, need that it was supposed to meet.- Where the product — ; 

■ . '*•..•■•'" .- * * '1 . • ' ^ . (I . ■■■.«' ' 

"as iif 'OUT* case — is a-compet^cy--based 'training-'ii^ this Icind of ' worry ^trpislates' 

itself into worries -about whether one Can view- the little skill that was acquired 

ras-'.g^ieralxzable to th^ji many other*.-: aimlar' s]d.lis tJ^^^^ be needed in . the. 

\. ■ ♦ ' • classroom. ' ' ■ — . -.^ ' 

i: ■ • ; " ' - ■■ : ^ ^ ^ ^ 

■i " . .... . 3 . 12 J Cost Effectiveness ' , " - 



V We now have to look into questions of cost, and a very complicated business that . 

' is .indeed.. ' The use of a C.P^A. is not enough; the use of an economist is not . 

* ' ■ ■ * . . \ - " . ' ■ ■ ■ ' ' ■ ■ •• ■ " , 

enough;* we usually have to use an economist with special expertise in the. field ■. ^ 
- ■ - "• * ■ " ' ■ • ■ * " ■ ■. • ■ . 

• school, economics; and a C.P.A. with special, expertise in the .various^ types of . ^ 

bookkeeping- u^ed by schools at j^imary,. secondaryo and tertiary levels. • • " . ' 

■• ; ' . • ■'• -.■ ■•■ ■ 

. ' When .we -have- adequate data on the^^^ the cppcrcunity, the. direct and the indirecit, 

• (the discounted cash flow, land other costs* and arialys^V^tl^ we can combine them 

ERIC". . 'r ■ . : 
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■ wilii the results* of , ched^ 11 tg yield eiliier a cost-benef it^anolysis 
(at least) or a cost-effectiveness ^analysisr^^ most cases where we, find 

■ reasonably, sijnple diinehslohih^ ' ■ 



- .3.13 ' Continuing Support 



\ 



This is a funny little iten, as b^its Jits nuniber and its location on the /• 
checklist.. But ;it ' s^soinething' of the sting in the tail of the ,checklist, too. 
It <x>ncerns the /question V of avoiding orphan^, and j&/hen we start, looking at orphans 
in the educational iiarketplace , we are ^ixaich^. struck by the fact that the little' 
entities that everybc3dy wanted to be heir apparent s to on* one day are often' ■ 
entirely orphaned the next . / Faddism is the great enemy of educational progress V [ 
and in^ order- tp avoid faddism, we need, in the oqurse of a serious evaliiation, ^ , 
to look at; the ques'tion of siistadned "support -and ix^^ giv^ educational 

prograiTi, project, or product. Will Sdiy- stop manufacturing video cassette recorders 
in the near future, just after we've |>urchased, a set -of th^ for the school? Will,y . t 
the behavioral objectives mDvenient indfeed be steadily amplifying the libraries or" ■ 

■ ■ • •■ ' '-A - ■ ■ ■ : \ . ■ . ^ ./-'^ 

instructional objectivies after we conveart.. to'-^ commitment .to behavioral objectives?^ 
Will -there be' prograiiro^ that thelstudents can use once "they're .thro\igh this*, 

set of programmed texts , "or will' they be_pr^ Will the competency- 



based novenient/ follow through?" Ihe ■ evaluator has to investigate such questions. 



iv,; -The Specific Problem of Teacher EvalA:3:afeQn. 



^ I willTsimply mention 15ie f act that, apart f ran "die pr6blCTi of evaluating competency- 
: ^^^^sed teacih.er education, tinere .is also the *'j^blem of evaluating the problem of ' • 
' eS^^^iTig' the products of CBIE programs , , name^^ / . 

OT' re'"'we run "into a number of other* quite cBffereSt problems for .the evaiuator* 
"■B»w,dp you get' from^'- a oxiclusion ato ixppacted population 



of having corripetency-based teachers to the inerit of »an individual teacher? Tnere 

is no easy path. Even such- sinple quest ion§ as how one gets fxx^m corapetengy-nastery 

' ^ • ' ' . ■ ' ' ■ ■ • > ... 

j^adgrna^ts ^Cwhich one inakes on the course of . supervising the trainees ) to cqmpetency- 

. exeroisa ^conclusions (which is what one needs in order to be sure that his coii5>etencies. 
w:l11 lie used in the- classroom) ? • Jan^s Pophan has clearly stressed the important' 
distinction between compet^cy-b^sed approaches to program evaluation and • 

.competency-based approach<§s ta individual teacher evaluation; *giv<en the present 
instability of data on ccmpetencV of "ceachers as individuals, we by no means 
have to* throw up our hands about the possxbility'of evaluating training programs' 
using" canpetency-based tests- Again, there .ds the question of whether we 'are 
goirtg to evaluate teachers for minimum cbmpetency'i^ or for purposes of developing a . 
competency profile on 'which one could rely for, e.i^. /placing .a. specialist. --^^T'te ^ 

former is in muchh^etter shape, as the evaluation .^rt stands, , than the latter. 

• ■ ' ■ • ■ . ' y ' - 

Li short 5" v/hile the conipetency-based approach offers us considerable opportunities " 
■ for improving the present state of teacher evaluation, this mainly shews that 
the pr^esent statS of teach^ evaluation is in-v^ry bad shape, not that the 

■ ' ' .. . ^ • ' . ■ , ■■ "if , . , " . ■ •" ■ . 

competency-based approach -is -without its own problenis. 

' ■ ■ . " — . ■ . ' ■ . . ■ • ] . 

•5.. dost-free Evaluation - ■ = : 



I promised to add a little touch of gopd news to the end of .this -list of worries,^ 
ar^d I wish there were time to develop the;' good, news at comparable length! But 
I dan at least explain the general thesis tl'ikt I want, to propose 5 even if T can't 
get into details, about the exact .re^ons for believing it to be ;^e. . . " 

:ihe general "thesis is that the costing of evaluation itself is, usually incorrectly 

"■■ ■ • . '.■ V •■' ■. • ^ ■ ' 

^ -nej; it , is usually -seen, as a cost on the .books of either ah agency or-a prpject^ 



. ■"15- ■ . • 

or perhaps a. school distriGt, arid the entry thex^ reflects, the fact that somebody 

• has , to iriake out' a chedc to pay the 'eval;;iator.' What ''d^sn' t«^how up there'' is the 
hidden side of the ledger, namsly, the returns from -die evaluation, Exactly the 
.saine poiiit can' be niade about- a fire department.^ If you ask the nayox^ of a .city 
what- the cost of the fire department is, iJaen.he or she is l^ely to reply by ... 
'.irai>tioning the figure that i?epresfents the airount of ironey paid oVer . by the 'City 
. Treasurer into the fire department account each year. Of course, the fire department 

isn't just a cost. \.The department's existence is what keeps insurance rates down 
jjm the city. .The department ' s-^existence is Vhat keeps the death rate and property^ . 
' damage rate down iri) the city. The returns from those activities^of the department 
have to be taken ii;to account in a correct set of social books ^ The same is true 
, of evaluation. Evaluation is not a tack-on luxury, stuffed down our throats by ■ V 
external agencies. It is what it takes to determine the. qiiality of one's own" 
work. V/ithout effective evaluation, there can be^ no knowledge that what has been' 
done has ' been worthwhile . If , therefore , one ' s commitment is - to worthwhile 
activity, then, one is necessarily- ccmmitted to substantial evaluation.- ^ Now, 
if we start drawirig up a set of social books y then we have to reco^ize that 
evaluation is frequently responsible for terminating projects that are not * 
worthwhile i' and it is somf^tioies responsible for increasing the amounts of ^ r-/ .] ■ 
r esources that are put iiito pther p roject s" of exceptional promise, tl^reby getting 

: very\ large scale payoffs. Evaltlation/ in. short., . 

iOuld not represent a net .cost to a community, such, as,;, the teadnejr educators. 

■ ■ ' ' ■■ i' ' \' ■ -v ■ ' ' ' ' 

. I vion*t say that bad .evaluajtion don't cost more: than they shouldy^' by my standards, 

that's an excellent ground fbr saying that^they' re bad, ""^ince" they fail to out- 

berform a critical competitor. I only say that a good evaluation is not only a 

• good- investment but the onlyl means. wherCby you can recognise whether, your .investment 

■ •- " ' ] • ■ •' "■• ' ■ • ' . ^ .. " . . 

O 3- good or not. So, if I have presented a rather .(impressing list of activities that 

ERJC . . " " . j ■ ■W'^-- /••'■■■ 

satisfactory.. evaluation, and thaf are not taken ciar^yof.by the nov^.^ 



them beyond the' threshold of 

' 4 



to cornperency-based approaches, I have, least- provided some Jdiid of a sugar 
coating for the pill that I am suggesting^. one has to swallow! " 

6 . Conclusion ■ . . * " • . 



Coiapetency-based -evaluation' is a" Jiieans to- the ends of education. Whether it 

" • . ■ . .»..■'• * 

iSecaT^es one inore seniantic taTistone in the" 'deiiietary of educational fads depends 
• entirely on two issues; first, whether we can keep, our eye on ^ (Quality and quality 
conxrol, and ^eet-the standards of quality, not mere popularity; §econd, whether ' 
we can ^ct like rational users or consuiners. and adopt on the basis of inerit'. 
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